Time-Sharing Policies for Controlled Markov Chains

We propose a class of non-stationary policies called \policy time sharing" (p.t.s.), which possess several desirable properties for problems where the criteria are of the average-cost type; an optimal policy exists within this class, the computation of optimal policies is straightforward, and the implementation of this policy is easy. While in the nite state case stationary policies are also known to share these properties, the new policies are much more exible, in the sense that they can be applied to solve adaptive problems, and that they suggest new ways to incorporate the particular structure of the problem at hand into the derivation of optimal policies. In addition, they provide insight into the pathwise-structure of Controlled Markov chains. To use p.t.s. policies one alternates between the use of several stationary deterministic policies, switching when reaching some predetermined state. In some (countable state) cases optimal solutions of the policy time sharing type are available and easy to compute, whereas optimal stationary policies are not available. Examples illustrating the last point and the usefulness of the new approach are discussed, involving constrained optimization problems with countable state space or compact action space.